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ABSTRACT 

This report presents a plan for ongoing evaluation of 
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The curriculum is an important departure in the University's approach 
to the education of undergraduates, it provides an option for those 
students whose aims tend to te action-^oriented. It is generally 
concerned with their competence as decisionmakers and ef fectuators, 
and is directed mainly to those intending to enter graduate 
professional training. The report describes the performance 
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information; an abstract model of the curriculum and a corresponding 
picttjre of the way it is being actualized this year; behavioral 
objectives for students in the program; forms for collecting 
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advisors; a computer printout of that part of student evaluation of 
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to those intending to enter graduate professional training, inasmuch 
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constantly changing, 't Is important for the curriculum to be constantly 
in process of evaluation and redesign, it is for this reason that the 
evaluation plan takes on special significance. 
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information flows of evaluative Information; an abstract model of the 
curriculum and a corresponding picture of the way It Is being actual- 
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PREFACE 



'The Professional Competence Core— An Interdisciplinary Curriculum for 
Undergraduates" is also known as "The Creative Problem-Solving 
Curriculum for Undergraduates." The second title is used throughout 
this report. The report describes how the program is to be evaluated. 

The curriculum has been initiated, developed, and guided by the deans 
of seven of the graduate professional schools at UCLA. They are: 

Lester Breslow, and former Dean Lenor S. Goerke, Public Health 
Maurice Connery, Social Welfare 
John Good lad, Education 

Harvey Perloff, Architecture and Urban Planning 
Hurrr-^y Schwartz, Law 

Chauncey Starr, Engineering and Applied Science 
Harold Williams^« Graduate School of Management 

On behalf of all those involved in the program, the principal investi* 
gator expresses deep appreciation for the vision which led the deans to 
sponsor the program, the good will they have shown in its formative 
stages, and the substantial contributions they have made to its imple- 
mentation. Appreciation is also due Deans Kenneth Trueblood of the 
College of Letters and Science, and Charles Speroni of the College of 
Fine Arts, who have joined the group of graduate school deans in help- 
ing to move the program in desirable directions. 

The curriculum is intended to help prepare students to shoulder the 
responsibilities of professional and similar roles in a more competent, 
confident, and mature manner than has been typically possible hereto- 
fore. It is felt that tnis process should start at the undergraduate 
level if not before, and that 'I requires augmenting the education of 
undergraduates with experiences that are process-oriented (as di:>tin- 
guished from substance-oriented courses). The program spans most of 
the colleges and schools on the Los Angeles campus. It also spans 
the gap between undergraduate and graduate levels. 

in these respects, we believe it is different from programs with 
similar objectives elsewhere. These features have accounted for much 
of its strength, but they have also contributed to the difficulty of 
"collective design." We would like to thank all of the students and 
faculty members who have contributed ideas, work, and patience to 
develop this evaluation plan. 



( 



TABLE OF COMTEMTS 



Abstract , I 

Title Page 2 

Preface [3 

Tabic of Contents , \ /...[.. k 

List of Tables ; 5 

List of Appendices , 5 

Introduction , 5 

1. Background of the Curriculum 6 

2. Statement of the Problem 9 

3. Tasks 10 

Methods and Procedures I) 

Results ^ !.!!!!!!!.!i2 

Task 1. Design Information System, 12 

Task 2. Modify Curriculum Content and Methods and 
Task 8. Generate Curriculum Model to Show interrelation- 
ships Among Courses/Modules 33 

Task 3. Behavioral Objectives^. 36 

Task k. Develop Student-Attitude-Measuring Instruments, 
Task 6. Develop Instruments to Measure Student Attitudes 

Toward University Education and 
Task 11. Pilot Use of Instruments with Participating 

Students. ^ (^8 

Task 5. Devise Forms for Faculty Members 50 

Task 7. Generate an Advisory Group for the Program 53 

Task. 9. Develop a Factor-Analytic Procedure Based on 

Q-Technique 57 

Task 10. Develop a Statistical Design 57 

Conclusions 53 



LIST OF TABLES 



Table 1 Program Performance Objectives Plan 

Table 2 Information Points Key 

Table 3 Schematic Curriculum in Creative Problem-Solving 

Tabic k Actual Curriculum in Creative Problem-Solving 

Table 5 Behavioral Objectives 

Table 6 Instructor's Course Evaluation Form 

I able 7 Program Advisory Group 

Table 8 Advisory Board Member's Program Evaluation Form 



LIST OF APPENDICES 



Appendix 1 
Appendix 2 
Appendix 3 



Appendix k 
Appendix 5 



Student Preliminary Data Sheet 

Student's Course Evaluation Form 

Table 1. Key to Student Course Evaluation 

Computer Printout Summary 
Table 2. Student Course Evaluation 

Computer Printout Summary 
Factor Analysis Study Report 

Selected Scales From Higher Education Evaluation 
Kit 



INTRODUCTION 



J^obll^r? r«P'""«"55 f for ongoing evaluation of the Creative 

i. al In Ir"? Undergraduates at UCLA. The curr culum 

o u d "Sr^duatJ"'^ ""-erslty's approach to the educat on 

tend to L ^^J- ' '^T'**" option for those students whose aims 

tend to be act lon-or.ented. It is generally concerned with their 

^n^irtntend-n'n' ef fectuators . and Is diJected ;:«inly 

to those intending to enter graduate professional training. Inasmuch 

co^J^^m! changing it is important for the curriculum to be 
constantly in process of evaluation and redesign. It is for this 
reason that the evaluation plan takes on special significance 

I. Background of the curriculum. 

IchiSts'^irenirin"' ^^^"""^ "^'"'^ S'-^**"^^* professional 

of ^!^;n!t-'^ Z*'*^'"!'* proposals for the creation and development 
oJieited Mr- r*'«'-9:f<'"^t« Program which would be valuable to act on- 
oriented, decision-making future professionals. 

cllurim^'' f independent sub-committee of UCLA's Goals 

erluon. ' ' experimental curricula in undergraduate 



even of 'of^5^'"1'''r "^'■^^y ' ^l""^ °f 

a Io?nf JJ^r? ? 3 ^° the possibility of 

Ir'S^ J / T ^ undergraduate program, focused on the 

problem-centered approach to education. 

IJudii^! ^" 3«ion-oriented curriculum alternative for 

students contemplating entry into a profession, while at the same time 
providing a needed innovation in undergraduate education. 

DltnarrfJufr"?** the preliminary concept and appointed an interdisci 
pi I nary faculty to work it through. 

Sfer^Jrn^r**** i^^' professional schools would 

tauoht hv c^?'■^'^^"^'". ^'"'■'"^ ^ sequence of "Momentum Courses/ 

evalJlt?L T^'^* <^°"'-^« enabled ' 

coulc^ !^ °" ^" e^Pe'-'"«ntal scale, about the individual 

oronr^m'H- '"^'^^'P^ting faculty, the student constituency, and the 
S?OhM-?'T"^"- '^^'^'^tlonally. the first momentum courses gave 
on nd r,Ji ' "tablished its reality, started students 

and a^ ?!- • ^^P^l"!*""^' a"** <^'-eated a process around which ideas 
o??a?^^r ^'^"'i systematically develop. While the program was 
preoara L jr''"** ^J^' ^''"^''^y ^'^<^ at. pre-profess ional 
ThTunl L \ 't presently has a wider participation and appeal. 
Through a planning grant received from the N«tlonal Endowment for 
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Dean. School of Architecture and Urban Planning 
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the Humanities, hieetings and work sessions with outstanding faculty 
from the College of Letters and Science and the College of Fine Arts 
have been held, to search for methods and arrangements to integrate 
the humanities into the program. These meetings are continuing 
through September 1972. A report of the issues and recommendations 
which emerge from these intense interdisciplinary work sessions will 
be prepared in December 1972. 

The Advanced Momentum Year 1972-739 will offer undergraduates an 
expanded set of courses which cover more of the categories in the 
Curriculum Model, and are taught by an enlarged cadre of faculty 
drawn from a broader base of participating Schools and Colleges. 

Universities tend to conceive of undergraduate education in terms of 
academic departments. Degree requirements tend to refer to combina- 
tions of courses drawn from those disciplines. Students, on the other 
hand, frequently have educational objectives which do not map well 
into those departments. 'The bridge from knowledge to action tends 
to be incompletely implemented. 

The CPSC is an attempt to create one such bridge. The CPSC does 
not replace a standard academic major. It is an adjunct to a major, 
providing students with an organized, goal-oriented suite of courses 
for satisfying some of their "breadth requirements" and electives. 
They are not relieved of the need for rigorous satisfaction of 
disciplinary requirements. 

Students in the program may come from all majors within the university 
While at the moment, any student admitted to the program may take any 
number of courses within it, we expect to set a minimum of three as 
soon as administrative arrangements can be made. Eight is expected 
to be a practical maximum. The specific choices, and the sequence 
in which they are to be taken will depend upon the student's back- 
ground and interests, as well as the designed relationships among 
sets of courses. It is expected and intended that, generally speak- 
ing, those who opt for this program will be among the best students 
in each field. If "best" is well defined, the program should also 
serve some of those whose academic records may not, for various 
reasons, be outstanding, but who are strongly motivated, intel lectual 1 
we 1 1 -endowed, and willing to follow appropriate paths through the 
complex domain of the program offerings. 

Some of the courses are devised especially for the program. Others 
are modified from present catalog offerings. Inclusion of a course 
depends upon agreement of the instructor and his department, not only 
that he will orient his course toward program objectives, but will 
participate in the subsequent design of the program as part of the 
program faculty. Students who take courses in the program are also 
required to meet with other students and faculty in the continuing 
evolution of the curriculum. In addition, both faculty and students 



must agree to provide feedback information on courses and on the program 
as a whole, on which the directions for change are partially based. 

Students toward whom the curriculum is aimed wish to move toward social 
roles that will exert considerable influence on the interests of others. 
The general character desired of incumbents of such roles » as judged 
by the originators of this program tend to include terms such as the 
fol lowing: 

Student should be able to understand, feel comfortable with, and operate 
effectively In an environment characterized by: 

Large scale of commitments, investments, threats, consequences. 
Uncertainty and ambiguity in important decision situations. 
Complexity and interaction of aspects of the world, including those 

at various levels of discourse. 
Rapid change, including change of attitudes and aspirations of 

people, political structures, technological developments, and 

envi ronmental events. 
Multiplicity of values and kinds of power. 

High degree of organization and institutionalization of interests 
and means. 

A need to anticipate future states of affairs and the consequences 

of present actions and decisions. 
The appearance of unexpected opportunities. 
High rates of information transfer and knowledge generation. 
Sustained importance of science and technology. 
Attempts on many sides at manipulation of thought and behavior 

He should: 

Understand and appreciate diverse values and value systems. 
Learn to learn and to think more effectively. 

Become better able to use the established as well as newly available 

intellectual and other tools. 
Understand how the system works at a number of levels and how to use 

it for his own and others' ends. 
Understand how to change the system, the extent to which it can and 

will respond to attempts to change it, and the limits to its 

changeabi I i ty . 

Understand the implications of various kinds of balance between 

stability and change. 
Understand and be comfortable with the need for accountability in 

an open system. 

Not experience undue anxiety as he approaches progressively more 

demand i ng cha 1 1 enges . 
Be oriented to achieve and contribute to society somehow. 
Be able to integrate his experiences into meaningful wholes which go 

beyond the elements. 
Be able to communicate effectively with others-- i ncl ud ing members of 

groups which are different from him ethnically, socio-economical - 

ly, professionally, etc. 
Enjoy and be able to profit from practical experience, and be 
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oriented to apply his knowledge and prior experience. 
Strike a practical balance between cooperat iveness and competitive- 
ness. 

(Note that the "content-free" appearance of the above objectives is 
qualified by the association of the curriculum with a standard "major" 
in all cases. Moreover, these objectives will be achieved through 
studies and projects on particular issues which "bring them down to 
earth." Decidedly, however, the curriculum is "process-oriented.") 

What do terms such as those above mean in operation? How are such 
desiderata to be realized? That is the task of the curriculum design 
project. It requires inculcating skills in the use of intellectual 
tools and techniques, the generation of a degree of emotional maturity, 
self-awareness, and readiness to assume responsibility, plus exposure 
to practical affairs of the proper kind In a graded, progressive way. 
While such matters are not universally perceived as the main business 
of the university, they arc completely consistent with its character. 
It is the business of the rest of this report to describe what *he 
discussion above implies in practical terms. 

2. Statement of the problem. 

The program as it is designed at any time represents a hypothesis 
about the set of learning experiences that will best prepare those 
who will later occupy positions of authority and responsibility. It 
does not, in its early stages, have the sanction of long development 
and recognition, which standard courses and curricula do. What it 
does have behind it is a sound concept, a careful design effort which 
is broadly based across disciplines, a serious, systematic, and 
evolutionary approach, a commitment to quality, and an affirmative 
response from students so far. Its long-term value, however, is yet 
to be established empirically. Consequently, a conscientious program 
of evaluation and feedback is essential. The function of the evalua- 
tion program is to provide an ongoing test of the hypothesis. 

The spirit of the evaluation plan is contrary to the "go — no go" 
approach that is sometimes attached to the term, "evaluation." The 
objective is not to decide whether to accept or reject the program, 
but to use the best resources available to create a program deemed 
likely to have the desired result, and then to use feedback to diag- 
nose problems, and make the changes necessary to achieve that result 
better. 

We perceive the need for a multi-indicator, diagnostic approach. 
"Mult I- Indicator" because no single figure of merit is sufficient 
for the various purposes for which the different interested parties 
will need the information; "diagnostic" because we know at the outset 
that there must be a discrepancy between what we can now offer--with 
very limited resources and severe time stringencies— and that which we 
would ideally offer to meet the challenge. 
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The curriculum, as an educational innovation and subsaquently as an 
i ist i tut ional ized development^ is responsible and accountable to the 
students whc participate In it; the faculty members who organize and 
teach in it; the university administration which supports it; the rest 
cf the university community^ of whose institution it becomes a part; 
other institutions of higher education^ which may have some interest 
in adapting some of its features for their own use; and the community 
of actual and potential sponsors for portions of it, including the 
people of the state^ the U.S. Office of Education, the National Endow- 
ments for the Humanities and the Arts, private foundations, and corpor- 
ate and private donors. Hence it re^ ' « administrative arrangement 
(preferably a simple one) to attend s -..larous interfaces. That 

administration, no matter how streami i ..^ j, will require a means for 
gathering and organizing the information it needs for its own decision- 
making, to be described below» and for reporting to others. 

The many potentially interested parties are indicated in Table 2 . 
Each different group or person is likely to assign a different weight 
to each of a variety of measures of performance. Hence we have an 
obligation to supply the various measures so that they may be combined 
to suit each purpose. There is neither a need for, nor a useful 
possibility of^ a single satisfactory scalar to sum up the program's 
va 1 ue . 

3. Tasks 

The project consists of the following eleven tasks identified in the 
proposal : 

I. The design of a simple information system for the collection, 
organization and accessing of both short and longer-term data relevant 
to program evaluation. 

2* Modification of program contents and approaches in line with 
continuing analysis of the over-all program objectives. 

3. Development of behavioral objectives having implications for 
the design of courses and modules, and for their sequencing; and 
generation of appropriate measures of performance. 

4. Design of in::^truments to measure student attitudes towards 
the program and the instructors. 

5. Design of instruments to record faculty members' observations 
about the program. 

6. Design of a form for student evaluation of university education. 

?• Formation of an advisory council of experts, both on this campus 
and elsewhere, and the devising of procedures to solicit and systematic- 
ally utilize their advice and criticism. 
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8. Generation of a curriculum ''moder' to show relationships 
among behavioral objectives. 

9. Development of a factor analytic procedure based on Q-technique 
to interrelate in meaningful ways the diverse behavioral a^d dttitudinal 
changes that may occur, leading to testable hypotheses for further 
research. 

10. Selection of a statistical design to identify sources of 
observed variability. 

11. Possible use, on a pilot basis, of these instruments with 
students who have already participated to some extent in the program. 

Work and results on these tasks will be discussed in the order given 
above. Conclusions on the program and its evaluation as a whole will 
be discussed at the end. 

METHODS AND PROCEDURES 

This was a design and research planning effort, so that the majority 
of the work was synthetic rather than analytical. " 

1. Students and professors participating in the program were 
asked to discuss the needs for and design of the curriculum. Notes 
on these discussions were transformed into progressively improved 
versions of the curriculum model, evaluation variables, and course 
ideas. 

2. Individual faculty members from all parts of the campus were 
concacted i>ased on their reputation among students nnd faculty for 
innovative and effective approaches to education. They were invited 

to make suggestions about various aspects of the program, and to suggest 
courses, methods, and arrangements that would improve the curriculum. 
Their views on measurement and evaluation were also solicited. 

i. Several meetings of the Deans of the Graduate Professional 
Schools were held. Other individuals were invited to these meetings. 
The discussions clarified the intent of the sponsors, and enabled new 
ideas to be presented. 

k. A preliminary analysis of currently offered courses in the 
program was transformed into a partial list of performance criteria. 
This was then expanded on the basis of the stated objectives to pro- 
duce the table given in the text. (Table 5) 

5* Several versions of a student data form, and student evaluation 
forms were constructed and tried. These were based on a) the need to 
have baseline data for later use in comparisons, and b) information 
from earlier efforts, both on this campus and elsewhere, as to the 
kinds of items that did and did not provide dependable information. 
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6. Expert consultants were used to provide guidance on approaches 
to and instruments for making the kinds of measurements that are 
appropriate to this scale and kind of educational effort. 

7. Extensive research was conducted into related programs and 
the methods they use for evaluation. 

8. A search was made for measuring instruments to cover the 
variety of items we deem important to measure. This was done through 
published sources, and through professional contacts. 

9. An analysis was made of factor analytic programs available 
on this campus to determine which ones to apply to the kinds of data 
we can expect. The pro's and con's of each program were compared 

to produce the strategy given in the text. 

10. A computer program was written to process the objective 
portion of the student evaluation forms. Analyses were run on eight 
of the courses offered last year for which adequate data were 

avai lable. 

11. A survey was made of last year's teaching faculty to determine 
the kinds of behavioral changes they expect their courses to make if 
offered again. 

12. Discussions were held with potential advisers to the program, 
to solicit their participation on the AJvisory Board. 



RESULTS 

Task I. Design Information System. 

A program of the scope and complexity of the Creative Problem-Solving 
Curriculum for Undergraduates requires constant and careful navigation 
and management. The needed management must rely upon a well designed 
and conscientiously maintained evaluation program. Task I of this 
project has been to develop an information system to support that 
program. 

The most important distinction to drav/, and one that is very elusive, 
is that between program evaluation information and course evaluation 
information. The program as a whole is intended to accomplish ertain 
things; each course attempts to contribute to the satisfaction of 
selected program objectives. Among them, there will be overlap and 
redundancy. There will also be thin places, where objectives are not 
as well attended to as they should be. But the program as a whole will 
accomplish things that no course can: for example, it may change 
student attitudes toward the university; it may build the student's 
confidence that he can handle problems of various magnitudes; it may 
enable the student to draw generalizations of a higher order than any 
single course would; it can enable the formation of working groups of 



students; It can institutionalize a new approach to undergraduate 
education, etc. Hence the program is not Identical with the sum of 
the courses. 



The first order of business is to get clear on what the basis will be 
for program evaluation. The answer, as of now. Is given In Table I. 
That table is arranged not in order of Importance, but in chronological 
order, starting with admission and selection of students into graduate 
professional school, and going on to the more Important, but perhaps 
less xJIrectly measurable Indicators of program performance. When 
this Information is augmented by an indication of where the data are 
to come from, and how frequently, and to whom and how frequently they 
are to be disseminated. It constitutes the major part of the information 
systui description. The remaining parts are associated with identifi- 
cation of course performance objectives and how they will be judged 
In terms of performance measures as given below in Table 5 ; the base- 
line, or "before" data on students, which are routinely collected at 
the time a student first applies for admission to the program or one 
of its courses, using the form given in Appendix I; and the various 
other pieces of information described in the subsequent sections of 
this report. 

A few comments are needed as a guide to the interpretation of Table 1. 

I. A list of the potentJally interested recipients of information 
about program performance is given in Table 2 . The numbers associated 
with these recipients ar;. the ones used In the last column of Table I 
as a concise description of the initial dIstribuMon plan. 

A number of the criteria used In Table I are of the form (X-^)kO, 
versus the alternative hypothesis (X-Jf)<0. The "-" In the form of 
this hypothesis does not inply that we are shooting for no difference 
or effect; naturally, the greater the difference in the desirable 
direction the better. But we take the present level of university 
performance as the minimal standard which the new program must at 
least match. The (X-X)iO is a one-tailed test under the assumption 
that desired differences, although they may be important, may not be 
detectable on small samples in the short run. 

Note that the measures implied In the table are of two kinds: objec- 
tively measurable, and subject to judgment of competent judges. 
Normally, for the second type, the judges are either self-evident, 
or they are explicitly identified in the table. 

It should be stated at once that while this table, and the rest of this 
report specify a large number of variables on which Information is 
needed for evaluation purposes, the ability to gather, organize, and 
disseminate this Information is partly an economic question. Some 
of the items are relatively easy to collect and disseminate. Others 
are very difficult and expensive. Hence the degree to which the plan 
can be Implemented will depend upon the financial support It becomes 
possible to generate. 
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In reading Table 1, keep in mind that the curriculum is Intended to 
have three classes of effect: on students, on the university, and on 
society. These are treated as sections 1, 2, and 3, respectively. 
The effects on students are of prime interest. The effects on the 
university are important instrumental ly, as they enable effects on 
students and on society, but they also have to do with the changing 
role of the university as a community, as well as its comprehensi bi 1 i ty 
and perhaps acceptability to its constituencies. The effects on society 
are both short- and long-term. To deal with the long-term effects is 
tenuous and to some extent pretentious. The kinds of societal effects 
wished for can be gleaned from the statements made about students and 
the university, as well as from the program rhetoric. To think that 
they would be measurable in a way that would make them attributable to 
this program strains credulity. Hence, the itemization having to do 
with societal effects is very attenuated, even though a great deal of 
thought went into the matter. 

The table is arranged so as to facilitate later implementation of the 
evaluation plan. That is, individual items of information are identi- 
fied sepr itely, and the modes of their combination described. How 
they are to be interpreted is also indicated, and in most cases, the 
action to be taken is associated with each possible outcome. This 
information system description is, as stc.ted earlier, subject to pro- 
gressive modification as experience i:; gained, and additional people 
have a chance to review it and make recopimendations. 
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TABLE 2 

Information Points Key 

1. Individual student file/dossier 

2. Individual CPSP transcript 
3« Computer file 

k. CPSP bulletin/information board 

5. Peer evaluations 

6. CPSP newsletter 

7- Graduate programs file 

8. Planned interview counsel lor 

9. Professor evaluation f\}6 

10. Participating students 

11. Non-participating students 

12. Participating faculty 

13* Non-participating faculty 

14. Involved Deans 

15. Funding sources 

16. Campus administrators 

17. Off campus educational institutions 

18. Industry, professional groups 

19. Key personalities in related fields 

20. UCLA committees 

21. CEO 

22. Identified student groups 

23. Course S professor evaluation office 

24. Government agencies 

25. Extended University 

26. Non*un iversi ty progran^ participants 

27. Program staff and administrators 

28. UCLA Alumni Association 

29. Vice Chancel lor» Innovative Programs Offi 

30. EXPO office (interns) 
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Tas^ 2. Modify Curriculum Content and Methods 

Task 8. Generate Curriculum Model to Show Interrelationships Among 
Courses/Modules 

These two tasks were combined during the course of the work. The 
molecular entries in the list of curriculum contents with which we were 
working at the outset of the project appeared to take on meaning pri- 
marily in the context of a) over-all program objectives, and b) course 
experiences which would embody them. The result of this effort is a 
distilling down, rather than an expansion of material quantity. The 
result is given as Tables 3 and k. These are, respectively a schematic 
representation of the curriculum as now conceived, and its present 
actual ization. 

Turning first to the schematic. Table 3. The student's experience 
should start with an introduction to patterns of problem-seeking and 
problem-solving, which lays a base for subsequent experiences. It 
should end with at least one practicum course in which he learns to 
apply what he has learned to a concrete situation of interest to him. 
(Note that further practicum may well be encountered in subsequent 
graduate school experiences.) This practicum is also useful as short- 
term evidence of the curriculum's effectiveness. In between are three 
clusters of courses: The Knowledge Base; Tools and Skills; and Personal 
and Group Approaches, as well as exposure to professional problem- 
solving approaches in various fields. The contents of these clusters 
are indicated to the right of the headings. With this brief introduction 
to the curriculum, the next section of this report details the behavioral 
objectives which have been identified so far. Taken together, these 
define what is intended in the curri cul urn mode 1 , so that no further 
explanation is offered here. Turning now to the Actual Curriculum, 
Table fifteen courses so far lined up for this year are "located,'* 
approximately, on the "map'* of the schematic curriculum. Course 
descriptions of these are not included here as they are outside the 
scope of this contract. However, a sense of what they include can be 
obtained from Table 5 where behavioral objectives are associated with 
each of the eight major categories which carry Roman numerals in the 
curriculum model chart. 
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Task 3. Behavioral Objectives 



We have seen in Table I what is expected of the curriculum as a whole. 
At this point, we must ask by what means those objectives are to be 
achieved. The general character of the implementation has already been 
conveyed by the curriculum descriptions given in Tables 3 and 4, but a 
more thorough indication is given In Table 5, which lists behavioral 
objectives for the courses, individually and collectively. While this 
list is extensive, it is not to be considered complete. No list of 
behavioral objectives can cover all of the intended outcomes of this 
curriculum (or perhaps any other, for that matter.) It is important 
to appreciate the operational point of view which leads to enumerating 
such objectives, but equally important to realize its limitations. 
Explicit behavioral objectives can orient the efforts of faculty 
members and others who might otherwise misunderstand or unnecessarily 
disagree about what is intended; but they do not exhaust what is in- 
tended, some of which may not be practically or usefully measurable by 
usual formal methods. But more to the point, one of the intentions 
of this curriculum is Individualization a process by which each per- 
son becomes more unique, in keeping with his own "virtu" (an ancient 
term referring to both his powers or abilities and also his special 
characteristics -- a term which perhaps has been missing from educa- 
tional dialogue and deserves to be revived.) Clearly, the result of 
such a process cannot be specified In advance. And the degree to which 
the judgment of the Individual should be weighted, compared to that 
of other observers, in determining the extent to which it has occurred, 
varies from case to case. 

Keeping In mind that not all desiderata will be readily measurable, 
we must still proceed with those which can be measured. As mentioned 
before, those listed in Table 5 constitute a complex hypothesis as to 
what will achieve the macro- or program objectives of Table I. That 
hypothesis Is only a first approximation: It is very difficult to gene- 
rate an adequate list of behavioral objectives. Instructors tend to 
be unpracticed at it. But it is our Mope that they will improve through 
this program, and that the results will evolve well. The "hypothesis" 
is subject to continual reexamination and revision^ based both on the 
evidence of earlier performance and on possibly changed conditions. 
The individual behavioral or micro*object Ives must be regarded as 
themselves variables, whose choice is subject to change as the macro- 
objectives are modifiedt Insights are gained about relevance of 
different kinds of material, and the relevant knowledge, tools, tech- 
niques, and expectations advance. The performance requirements listed 
In Table 5 do not Imply that all of the capabilities desired must be 
gained within the program nor that they are anything but performance 
minima. In connection with the mathematical Items, for example, 
students may demonstrate proficiency by examination or proof o prior 
courses of an appropriate kind, and either substitute other courses 
instead, or else go more deeply into the material. Those coming from 
fields not emphasizing mathematical skills will still be required to de- 
velop literacy ark! a modicum of facility with at least some mathematical 
too^s. 36 



Not all of the objectives listed map well into performance meecures, 
despite the effort that has gone into making them measurable. In 
general, they will have to be associated with measures by each in- 
structor whose course Impfnges on the objective, and be attuned to the 
situation of the course(s). However, an innovative scheme will be 
tried, the dossier record* and the "legacy." 

Briefly, the program office will organize a file of student dossiers. 
Each dossier will contain those products of each student by which he 
wishes his work in the program, or related to it, to be judged. It 
might contain models, mathematical derivations or proofs, proposals 
for projects, project final reports, photographs of work done, essays, 
or any other artifacts the student deems suitable. He may insert 
materials into the dossier at any time; he may also delete items as 
they are superseded by better work, or for any other reason. The 
material In the dossier will be releasable by the student at his 
discretion to any person or organization who has a need to make a 
judgment of his performance. It will be regularly used by the pro- 
gram as a source of evaluative feedback. 

In addition, students will be expected to leave a "legacy" to their 
successors. This, too, may take various forms. For example, they may 
develop a useful computer program; they may generate a questionnaire to 
gather a certain kind of Information; they may collect and organize a 
useful body of data; they may uncover an improvement In a problem-solving 
procedure; or they may simply have suggestions to make to other students 
about how to get the most out of a given course of set of courses. 
Whatever form It takes, their products should be useful to others in 
some way, and they will be prevailed upon to leave their products in 
good order for the jse of others. Hence, one of the things they may 
leave is an Improvement in the means of organizing and passing on such 
results. The growing body of such products should provide one basis for 
judging the value of the program, and of the individual's contribution to 
it. 
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TABLE 5: BEHAVIORAL OBJECTIVES 



Introduction & Overview 

1. Be able to discuss the objectives of the Creative Problem-Solving 
Curriculum, describing its differences from other university 
offerings, and establishing its relevance to his own objectives. 

2. Be able to make formal statements (hypotheses or generalizations) 
which are testable. 

3. Be able to formulate a problem in researchable form. 

k. Given a complex set of events, be able to discern some useful 

pattern in tha.D. Given a recognizable pattern in a set of events 
be able to construct a usef M generalization therefrom. 

5. Given problems of comparable difficulty and substantial 

complexity at the beginning and end of the course (program) to be 
solved in a fixed time, provide a much better solution at the end 
than at the beginning. 

"How the System Works'' 

1. Oe.nonstrate a thorough knowledge of the workings of at least one 
important system (e.g., a university, a political system, a 
community, an industrial firm, a bank, a government agency, etc.) 

2. Be able to state the functional components and processes of a 
given operational system, e.g., legal system; political system; 
social system; economic system; educational system? 

3. Show understanding of how deci .ions are actually made in 
organisations, and the forces impinging on them. Indicate how he 
would attempt to influence such a decision if he felt it was 
necessary. 

Be able to describe the overt and tacit information system of a 
commun i ty . 

5. Describe adequately how a community could go about becoming and 
staying better Informed than it now is about important matters 
that affect it*^ 'activities and its future development. 

6. Be able to make recommendations which would increase the 
effectiveness or efficiency of 3 given information system. 

7. Explain why profit is a socially desirable feature of business. 
Explain why It Is socially undesirable. Be able to make both 
cases effectively. 33 



8. Be able to describe the close relationship between the problems 
of individuals in society, and the problems of policy formation 
and implementation in government and private organizations. 

9. Show an appreciation of the values of stability and change by 
differentiating, 'n a given environment, between institutions or 
entities which it would be desirable to change markedly, and 
those it would be undesirable to change markedly. 

10. Show an appreciation of the values and costs associated wUh 
creating organizations to get things done. 

a) in an industrial context 

b) in a governmental context 

c) in a community context 

11. Understand and able to state how different roles (e.g., 
manager, accountant, client, investor, subcontractor) tend to 
lead to different models of a situation and hence possibly to 
different courses of action. 



Probcm-Solving Knowledge Base 

1. Given differing major political philosophies, write a creditable 
critique of the relation of their positions to a present 

soc ietal issue. 

2. Describe some of the moral and ethical foundations of decision 
and action applicable to a given situation and show their 
philosophical and historical derivations. 

3. Be able to identify and distinguish among important psychological 
classification systems (e.g., Maslow's, Freud's, Piaget's, etc.). 

4. Demonstrate an appreciation of the differences between his own 
culture and others which are substantially different with regard 
to an indicated area of activity. 

5. Be able to state and exemplify the costs paid by organizations to 
a) maintain themselves, and b) fulfill the personal objectives of 
thei r decision*makers (organizational pol i tics) . 

6. Be able to recognize and state how the needs of people in various 
kinds of communities affect design projects^ and how to take 
account of them. Be able to document at least one personal 
experience which dramatizes these dynamics. 

7* Be able to outline the development of the traditional viewpoint 
on a specific issue. 

8. Be able to state alternative viewpoints to the traditional one. 

9. Be able to state the reasons for divergent views. 

3a 



10. Be able to incorporate both in an original essay on the subject. 



11. Be able to state formulations or viewpoints toward several 
important social issues that he considers to be original with 
him. 

12. Be able to state the prospective value of his original views and 
be able to trace their genesis and development. 



Mathematical Tools and Skills 

1. Be able to use the tools of verification — evidence and logic — to 
determine whether to accept or reject a hypothesis as a basis for 
act ion . 

2. Understand both the power of data and models in analysis and 
design, and also their limitations as representations of 
*'reality." Be able to explain this on paper. 

3» Be able to answer correctly the following questions with respect 
to a given example of statistical evidence : 

a) was the evidence gathered properly? 

b) are the units compared actually comparable? 

c) are the statistics really indicative of what you want to 
know or learn? 

d) are the statistics presented In their most meaningful form? 

e) are the statistics complete? 

f) are the statistics reliable? 

g) how could the statistics be misleading? 

4. Be able to define and demonstrate understanding of the concepts: 

central tendency regression coeff ic ient 

mean expected value 

median law of large numbers 

node central limit theorem 

variance percenti le 

standard deviation spurious correlation 

test of significance representative 

correlation sample size in relation to 

error variabi I ity 

val idi ty stratification 

rel iabi I ity population 

significance level statistic 

confidence level parameter 

5. Explain the meaning and Importance of Bayes' theorem. 

6. Demonstrate a command of the basic 'deas and uses of these 
mathematical concepts: element, set, relation, subset, proper 
subset, product, sum, identity, vector, matrix, add, multiply, 
measure, function, etc. 



7. Be able to describe, didgram, and represent mathematically the 
concept of feedback. 

8. Distinguish between positive and negative feedback^, and 
exempt ify each. 

9. Be able to show how the concept of feedback appears importantly 
in at least three different professional fields. 

10. Write a linear program for a specified problem. 

11. Produce a "decision t. 3e** for a specified decision problem, and 
show how to "solve" it. 

12. Show literacy in at least one computer language. 

13* Show evidence of having successfully used the university computer. 

14. Be able to describe and identify the limits to the usefulness of 
the cost'benefit approach. 

15- Cost benefit considerations: Be able to identify the cost to 
society in terms of alternatives foregone of adopting a 
particular solution to a problem. 

a) identify the monetary outlays associated with a particular 
solution (specified) 

b) identify the second best alternative to which each of the 
resources could have been put 

Estimate what the given solution offers to society in the short, 
intermediate, and long run, including both the prospective 
benefits and disbenefits. 

16. Describe the concepts of extrapolation, prediction, and forecast* 
ing, and show how they can be used to improve the operation of at 
least three complex sociotechnical systems. 

!?• Be able to describe at least six methods or techniques useful for 
forecasting or futures-anticipation. Be able to apply at least 
two of them in a given situation. 

I8. Show an adequate familiarity with the concepts of the differen- 
tial and integral calculus. Given a real or hypothetical system, 
be able to represent dynamic processes with fundamental models 
based on differential equations, and be able to internalize the 
meaning of such a representation. 

19- Possess an adequate level of numeracy, the ability to perceive 
numbers in their meanings. To perceive numbers as descriptors, 
as adjectives, rather than as mysterious symbols to be manipu- 
lated. Be able to estimate common and uncommon magnitudes of 
physical things. Believe that math is often as easy way to 
describe something. 



20. Show an adequate familiarity with the rudiments of probability 
and probabilistic modeling. Given a real or hypothetical syrtem, 
be able to assign reasonable probability estimates to potential 
outcomes and use the estimates for other decision processes. 

21. Be able io understand the structure of logical arguments. Have 
an adequate familiarity with the basics of formal logic. Be able 
to understand arguments based on indirect demonstrations of truth, 
e.g., disproving the contrapositive. Understand and be able to 
use the concept of implication wi :hout causal implication: 
correlation does not imply causation. 



Non-Quantitative and Semi -Quantitative Tools S Skills 

1. Be able to define and explain the relevance of the terms 
"systems," subsystem, element, environment, complexity, related- 
ness, teleology, cybernetics, entropy. 

2. Be able to state, define, and explain the relevance of the 
terminology of problem*solving and systems analysis. 

3. Have gained practical experience in at least one major communica- 
tions medium (e.g., newspaper, radio, TV, theater, recording, 
filmi. 

k. Demonstrate command of basic economic concepts. Describe the 
relations between supply, demand, price, etc. of a commodity. 
Enumerate the elements of cost of production. 

Show how to de.ide when to replace a piece of equipment, (e.g., 
an automobile). Indicate the considerations involved, how to 
quantify them, and what information you need but don't have. 

6. Be comfortable reasoning by analogy, especially about the form of 
problems acress fields. Show ability to find useful analogies, 
and to detect dissimilarities and points of doubt in their 

appi icabi I i ty . 

7. Given a specific problem context, be able to discern the relevant 
information in a mass of facts containing a great deal of 
irrelevant information. 

8. Be able to sketch or otherwise graphically portray the relevant 
structure of a problem situation or system, 

a) to demonstrate understanding, and 

b) to communicate with others 

9. Given a project plan, write a credible budget. 

'0. Define and discuss the term "overhead" in a budgeting context. 

II. Given a project description, write a simple PERT chart (or 
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20. Show an adequate familiarity with the rudiments of probability 
and probabilistic modeling. Given a real or hypothetical system, 
be able to assign reasonable probability estimates to potential 
outcomes and use the estimates for other decision processes. 

21. Be able to understand th^ structure of logical arguments. Have 
an adequate familiarity with the basics of formal logic. Be able 
to understand arguments based on indirect demonstrations of truth, 
e.g., disproving the contraposi tive. Understand and be able to 
use the concept of implication without causal implication: 
correlation does not imply causation. 



Non-Quantitative and Semi -Quantitative Tools s Skills 

1. Be able to define and explain the relevance of the terms 
"systems," subsystem, element, environment, complexity, related- 
ness, teleology, cybernetics, entropy. 

2. Be able to state, define, and explain the relevance of the 
terminology of problem-solving and systems analysis. 

3. Have gained practical experience in at least one major communica- 
tions medium (e.g., newspaper, radio, TV, theater, recording, 
film). 

4. Demonstrate command of basic economic concepts. Describe the 
relations between supply, demand, price, etc. of a commodity. 
Enumerate the elements of cost of production. 

5. Show how to decide when to replace a piece of equipment, (e.g., 
an automobile). Indicate the considerations involved, how to 
quantify them, and what information you need but don't have. 

6. Be comfortable reasoning by analogy, especially about the form of 
problems acress fields. Show ability to find useful analogies, 
and to detect dissimilarities and points of doubt in their 

appi icabi I ity. 

7. Given b specific problem context, be able to discern the relevant 
information in a mass of facts containing a great deal of 
irrelevant information. 

8. Be able to sketch or otherwise graphically portray the relevant 
structure of a problem situation or system, 

a) to demonstrate understanding, and 

b) to communicate with others 

9. Given a project plan, write a credible budget. 

10. Define and discuss the term "overhead" In a budgeting context. 

11. Given a project description, write a simple PERT chart (or 



12. 



equivalent In some other system of critical-path prograrwni ng) 

Be able to describe the relationship of a given problem to a 
theoretical framework or to previous research. 



13. Be able to classify correctly a given instance under one or more 
of ceveral alternative, possibly applicable, rules, or categories. 

I'l. Be able to answer correctly the following questions with respect 
to a given example of deductive reasoning: 

a) are the premises well formed? 

b) are they val id? 

c) does the conclusion necessarily follow from the premises? 

15. Be able to answer correctly the following questions with respect 
to a given example of inductive reasoning : 

a) how many instances have been observed? 

b) is this a large or a small number? why? 

c) Are there any counterexamples? exceptions? 

d) are the Instances fairly representative of their class? 

e) have they been chosen to support a preconceived idea? 

f) can you construct an experimental situation which you think 
would produce a counterexample? 

g) what other evidence tends to support the probable validity of 
the gene ra 1 1 zat i on? 

16. Be able to acquit himself creditably in an adversary proceeding. 

17. Be able to answer correctly the following questions with respect 
to a given example of opinion evidence : 

a) is the authority an expert in this field? does he have to be, 
to be credible? 

b) Is the authority prejudiced? 

c) is the reference to the authority specific? (I.e., is the 
reference a citation or a paraphrase?) 

d) is the authority aware of the significance of his opinion? 

e) is the authority supported by factual evidence? 

f) is the citation taken out of context? how different might it 
be In context? 

18. Given a field with which he is unfamiliar at the outset. In a 
given period of time (e.g., I week) demonstrate a satisfactory 
level of familiarity with its general character, what is known in 
it, what Its methods are, and what relevance it has to some 
professional or social concern of the student (e.g., a given tax, 
the law on some subject, laser technology). 

19. Be able to partition a given complex problem in at least two 
good ways, and demonstrate how the solutions reached differ and 
resemble each other. 

^13 



20. Be able to "A-R-Q" 



Analyze a scenario by translating it into a statement that can 
be expressed and examined quantitatively; 

Resolve the analysis in terms of a policy or other decision: 

Qua] ify the decision in terms of possibly different results that 
could result from: 

a) a restatement of the model used to formualte the problem, or 

b) different estimation techniques, or 

c) imagination in considering previously unanticipated outcomes 
deriving from the decision 

This should all be done in a fixed period of time. 



Personal and Group Approaches 

1. Be able to state the effect of specified body processes upon 
mental control • 

2. Be able to state the effects of various nutritional habits on 
body processes. 

3. Be able to describe various methods for reducing anxiety and 
mental anguish. 

Be able to state several alternative methods for: 

clearing the mind 
focusing attention 
extending attention span 
increasing concentration 

5* Be able to state personally effective techniques for increasing 
or reducing perceptual awareness. 

6. Be able to employ techniques of association and grouping for 
remembering quantities of factual information. Hence, be able to 
remember more such material than non-program students. 

7. Be able to document at least two different experiences as a 
participant in a computer-based simulation exercise; describe the 
learning, if any, that occurred in each instance. 

8. Identify a major area of belief or value that has been substan- 
tially Influenced by the curriculum, and indicate precisely what 
the change has been. This can mean a change or a reinforcement, 
but the effect must be documented and explained. 

9. Be able to describe himself in terms of several differing 
psychological viewpoints. 



10. Document at least two important Instances in which he shared 

with others some information about himself and a problem he was 
having. Describe each event, indicate the feelings and emotions 
that accompanied it, and describe the learning or other result. 

n. Similarly describe at lesat two occasions on which other shared 
similar information with him, etc. ... 

12. Become increasingly tolerant of and comfortable with courses that 
are not highly structured, and material that is not highiy 
organized a priori , as measured by attitude survey and course 
evaluation forms. 

13. Be able to contribute at least one suggestion, that is considered 
important, to the evolution of this program. 

\k. Concerning thinking, memory, attention, cognition, be conversant 
with the psychological literature in these fields to a reasonable 
degree, and be able to describe some late theories of how these 
processes work, citing some of the evidence. 

15. Be able to describe some of the processes he uses to improve 
these functions in himself. 

16. Be able to describe his own limitations in these functions and 
what he is doing about them. 

17. Document participation in a formal role-playing exercise. 

18. Document at least one object, system, program, or process design 
for which he was principally responsible. Describe the motiva- 
tion and the rationale, as well as the result. 

19. Understand that individuals identify with multiple groups, 
communities, interests, etc., so that they experience actual or 
potential tncompati bi I i t ies among actions/decisions. 

20. Be able to state the influence of multi-level purposes (and other 
sources of intrapersonal conflicts) on decision behavior. 

21. Be able to state the effects of perceptual distortion and 
selectivity, bias, goal gradient, projection, and other sources 
of personalization of perception on judgment or decision-making. 
Be able to describe means for protecting against their ill 
effects. 

22. Be mentioned by at least one other student as outstanding in at 
least one aspect of decision-making or problem-solving. 

Problem-Solving Approaches in Various Fields 

I. Show acquaintance with a range of conceptual models applicable to 
his field of professional interest, and the assumptions, 
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capabilities, limitations, and resources avai lable wi th respect 
to each* 

2, Given the names of two professional fields, be able to describe 
how they are the same, and how they are different- The descrip- 
tions are to be non-trivial, and incliude comparison of methods 
and techniques used. 

3. Describe how computers are used in at least three different 
professional fields or fields of social concern (e.g., education, 
health professions, transportation, retailing, law enforcement, 

I inguistics) • 

^. Be able to state how it is sometimes useful to bridge the inter- 
disciplinary and interprofessional boundaries \n pursuit of the 
solution of a problem; be able to give examples which show that 
the interdisciplinary boundary involves sometimes the same 
problems formulated in several languages; sometimes the same 
language applied to different problems; to overcome one's own 
chauvinism in terms of learning new 'Manguoges.'' 

5. Be able to demonstrate that solving a problem Is frequently more 
than simply applying a rule or set of rules; that this is true of 
the law, business, education, public administration, etc. Give 
examples in at least two fields. 



VIM. A Focused Problem-Solving Experience (Bringing the above together 
into a practical experience - in one of four forms. [Required]): 

1. Describe at least one formal proposal ^or action ^r policy or 
program change that he originated, led, or partic.pated 
prominently in. Describe the outcome, and explain it. 

2. Be able to document at least one "substantial*' experience working 
as an employee or intern in a government, business, or other 
organization in a problem-solving capacity. 

3. Be able to produce evidence of having solved or worked productive- 
ly on the solution of at least two problems of moderate 
complexity and difficulty in different fields of application. 

Be able to organize and effectively direct a group of his peers 
in group projects. 

5. Given the description of a complex, problem-rich situation: 

a) be able to select those problems that should be worked on 
first, and justify his choice satisfactorily, 

b) be able to identify those problems that are amenable to 
amelioration or solution within the state of the art, 

c) be able to formulate at least one such problem in a wcy that 
makes the course of solution clear to others, 

d) be able to identify the issues inherent in the solution or 
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amelioration of the problem, 

be able to separate the. more from the less important parts 
of the problem^ 

be able to estimate the time and resources needed, 

make a convincing case for the benefits to be obtained via a 

stipulated solution/amelioration procedure, 

be able to establish the limits of goodness of the expected 

result and show the expected disbenefits, 

define the major contingencies in the situation, 

identify the various interests involved and the positions 

they might take with respect to the major design alternatives, 

identify at least one reasonable alternative formulation of 

the problem, 

list the design criteria to be satisfied, 

indicate the evaluation measures and procedures to be used. 
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Task k. Develop Student-Attitude-Measuring Instruments 



Task 6. Develop Instruments to Measure Student Attitudes Toward 
University Education 

Task 11. Pilot Use of Instruments with Participating Students 

These tasks were combined during the project. It became apparent that 
inundating students, even with their consent, with too many quesion- 
naires to fill out was unwise, so that to the extent possible, items 
relating to attitudes toward university education should be added to 
instruments intended to measure attitudes toward courses, instructors, 
and the program. 

Preliminary Data on Students 

Appendix I is the form that has been developed to gather Information 
about students upon their entry into the program. This form was 
constructed based upon a) a desire to learn about students as they enter 
the program, and b) a desire to give students a chance to ventilate 
their attitudes and ielf-views, and to show an interest in information 
about them that the kinds of forms they are usually asked to fill out 
do not explore, it is fairly extensive, and is used as a screening 
device in two wr.ys: 1) It is a slight obstacle, so that students who 
are not intereited enough in the program to fill it out turn away 
promptly; 2) it provides Information on which judgments may be made as 
to student*:' readiness for various parts of the curriculum, or for any 
part of ic. No single Item Is used by itself as a screening item, but 
a judgir^snt is made on the basis of the form as a whole. An interviewer, 
who meets each student before admission, having read the student's 
fo^m, can use his or her time well in discussing directions to be taken. 
The forms are also useful in detecting change or movement between entry 
Into the program and departure from it. 

Despite the length and complexity of the preliminary data sheet, it will 
continue to be used In its present form for some time, since it has 
proved useful so far. Student reactions to It seem generally favorable. 
It may have a slight discriminatory effect, by discouraging students who 
can't easily tolerate what appears to them to be administrative red 
tape; however, that does not seem to be as s<?rious a price to pay as not 
having the kind of information the forms provide. And we try to 
minimize any aversion by the language of the form itself. 

Student Evaluation of Courses/Instructors/Program 

The form that was developed for student evaluation of courses, 
instructors, and program variables Is given in Appendix 2. The form is 
relatively self-explanatory. Note that Part II lends itself to direct 
analysis using computer methods. The analysis has been done for eight 
of the courses given last year for which data were available. The 
summary results are given in Appendix 3. Table 1 In that Appendix 



gives the key to interpretation of the computer printout, while Table 2 
is a copy of that summary prin:out by course. The last page of that 
Table, on page 113, is a sample of the computer printout of information 
provided by each student for each course. (A single copy of a complete 
printout of both course and student summary information is transmitted 
with this report. Its bulk makes the desirability of submitting multiple 
copies questionable.) On page 113, note the plot of individual response 
pairs differences, which shows, for each of the 35 items, the disparity 
between desired and actual levels of performance, and demonstrates that 
this student found some features of the course exceeded his desired 
level, while the rest split about evenly in meeting it and falling short. 
Normally, the detailed informatfon on each item, and totals, would be 
used, both in adjusting courses, and in counseling students. Both the 
levels and disparities carry diagnostic information, as indicated in 
Table 1 of the main body of this report. 

So far, no tests of significance have been applied to these data. 
Certain conclusions are possible nevertheless: 

Generally speaking, students have high expectations of the program. 

For the most part, their actual experience is not far from their 
desired level, although it is generally somewhat lower. There are many 
cases, however, where actual ratings exceeded desired levels. 

From comparison of the several sections of one of the courses, it 
appears that students' ratings of a course may depend to a significant 
degree on the instructor, rather than on the course structure, object- 
ives, or textbook, which were constant. Hence, the procedure we have 
been using, of recruiting carefully selected faculty participants, is 
preferable to asking departments for courses, or specifying courses and 
having them manned by departments "as available.** 

Thus, evaluation to date, on this pilot basis, is encouraging, 
although there is room for improvement. 

The Center for the Study of Evaluation at UCLA has developed a Higher 
Educa^ ^n Measurement and Evaluation Kit, which contains brief student 
questionnaire forms for use by colleges and universities in various 
connections. Many of the forms refer tc variables that we are inter- 
ested in. They have the advantage that they are very brief, they are 
readily scorable, and most importantly, they have been standardized on 
both a UCLA and a national sample of students. They have the disadvan- 
tage, in some cases, of referring to situations which are not quite 
like ours, so some of them are not usable for our purposes. During 
this coming year, we will experiment with the use of certain of these 
scales in the program. 
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They are based on a very important departure in viewpoint toward 
evaluation of higher education: that student evaluation of instructor 
and courses, while desirable for some purposes, may tend to put 
instructors and administrators on the defensive, which is not necessar 
My desirable. The value of their materials is that they are existen- 
tial rather than judgmental, and constitute an important part of any 
plan of evaluation in our estimation. We intend to work with the 
Center to select the scales that have most applicability to this 
program. Tentative selections are given in Appendix 5. 



Tasl< 5. Devise forms for faculty members. 

Table 6 is the form devised to elicit the response of faculty members 
participating in the program. Note the emphasis on diagnosis and 
suggestions, plus the encouragement to scrutinize the expert-nce, and 

^ ''^^l'^^ 9*" """"se with the students'. A^o n. • - 
t»ie request for measurable indices. 
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TABLE 6 



Instructor's Course Evaluation Form - Creative Problem-Solving 
Curriculum 



Course : 
Section : 



Instructor : 



Date : 



I. In terms of the objectives stated at the outset of the course, how 
successful do you feel the course was (Circle one.) 



Very 
Unsuccessful 



Very 
Successful 



3 k 5 



8 



10 



2. On the whole, how do you think the students will rate this course? 
I2 3<»56789 10 



3. Vfhat were the principal determiners of how the course went; 
(Number, in order of importance, as many as apply.) 



( ) Your own preparation 

( ) Your grasp of the material 

( ) Your choice of format and 

procedure 
( ) Your personal style and 

Interaction 
( ) Your acquaintance with the 

rest of the program 
( ) Ambit iousness of the course 
( ) Scheduling, including 

length of term 



( ) 
( ) 
( ) 
( ) 

( y 
( ) 
( ) 



abi I i ties 
motivation 
diversity 
backg round 



in the 



Students^ 
Students' 
Students ' 
Students' 
prog rem 
Counsel 1 ing 

Availability of assistance 
Textbooks, media, and/or 
I ibrary services 



( ) Other (specify) 



Please comment on the three or four most important ones 
i. 



2. 
3. 
1#. 



Additional comments: 
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What would you do differently next time? 



5. What suggestions have you derived for the program as a whole? 



6. List the measurable changes you think may have occurred in the 
students as a result of this course, and indicate how you suggest 
each change might be measured in the future. 



7. List some measurable changes you think should be attended to by 
other parts of the curriculum that may not be occurring now, and 
indicate how you think they should be attendee to. Also please 
indicate the measures you would use to ascertain that they have 
occurred. 
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Tas!; 7. Generate an advisory group for the program. 

Table 7 lists the advisory board members who have agreed to serve. 
They have been chosen for their range of relevant experience and the 
quality of their judgment, as well as their expressed interest in the 
program. Inasmuch as we wish to use their services in a systematic 
way without imposing too greatly on their time, we have devised an 
Advisory Board Member's Eyaluaf:Ion Form (Table 8). In return for their 
efforts, and to facilitate them. Board members will receive documenta- 
tion on the program as it develops, summaries of evaluation reports, 
and occasional samples of student products. We plan to have annual 
meetings where they are brought up to date on developments and can 
participate in the process of evolution at first hand. 
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TABLE 7: PROGRAM ADVISORY GROUP* 



BIckner, Robert, Public Policy Research Organization, University of 
Cal Ifornia, I rvine 

Churchman, C. West, Research Philosopher, Space Sciences Laboratory 
and Professor of Business Administration, University of 
California, Berkeley. 

David, Henry, Executive Secretary, Division of Behavioral Sciences, 
National Academy of Sciences - National Research Council. 

Michael, Dor.ald N. , Professor of Psychology, Professor of Planning 

and Public Policy, and Program Director, Center for Research 
on Utilization of Scientific Knowledge of the Institute for 
Social Research, University of Michigan; Visiting Fellow, 
Institute of International Studies, University of California, 
Berkeley 

Mood, Alexander, Director, Public Policy Research Organization, Uni- 
versity of California, Irvine 

Pace, C. Robert, Professor of Education, University of California, 
Los Angeles 

Park, Rosemary, Professor of Education, University of California, 
Los Angeles 

Ratoosh, Phil, Professor of Psychology, San Francisco State University; 
faculty member, Wright Institute, Berkeley 

Schon, Donald A., President, Organization for Social and Technological 
Innovation, Professor, Massachusetts Institute of Technology 

Tribus, Myron, Vice President, Xerox Corporati'' 



Note: These people have kindly agreed to serve the program in an 
advisory capacity. Their agreement does not imply that they have seen 
or app/oved this report, nor that they have participated in the pro- 
gram CO this date. 

These advisors are In .addition to the following Deans at UCLA: the 
seven graduate professional school deans: Lester Breslow, Public 
Health, Maurice Connery, Social Welfare; John Goodlad, Education; 
Harvey Perloff, Architecture and Urban Planning; Charles Speroni, 
Fine Arts; Chauncey Starr, Engineering and Applied Science; Kenneth 
Trueblood, Letters and Science; Harold Williams, Graduate School of 
Management 
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TABLE 8 



Advisory Board Member s Program Evaluation Form *- Creative Problem* 
Solving Program 

Advisor's name: Date; 
Aff i 1 iation: 



Please review the curriculum objectives and suggest changes you may 
feel to be necessary or desirable. Please explain. 



2. Please review the course descriptions and comment upon how well, 
and how thoroughly, they apoear to meet the o! jectives. Suggest 
any addition, deletions, or modifications you )u1d like to see. 



3. Please review the evaluation results of the program so far. What 
conclusions do they suggest as to a) the effectiveness of the 
program; and b) desirable changes. 
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4. What additional information would you like to have about the 
program to facilitate the formulation of your own advice? 



5. Whom would you suggest we add to our advisory panel for the 
program? Why? 



6. To whom should we send information about the program? What kinds 
of information should be sent in each case, and what is the 
prospective recipient's reason for interest? 



What programs at other institutions might have experiences that 
would be useful for us to know about? 



8. Additional comments. Please feel free to comment at any length 
about any aspects of the program. 



56 



Task 9. Develop a factor-analytic procedure based on Q-Technlque. 

In this context, factor analysis may be useful in providing some 
underlying conceptual order to the profusion of interrelationships 
among the many variables involved. Another way to look at it is that 
It can simplify the domain, which at first sight appears extremely 
complex. There are, of course, a number of kinds of analysis that 
might be performed. Q-Technique deals with the intercorrelations 
among persons, so that the structure that emerges describes the 
population of persons rather than variables. 

Several computer program packages exist on this campus to perform 
factor analysis. The two in most common use are the BioMedical Division 
(BHD) program system, and the Statistical Package for the Social 
Sciences (SPSS). Appendix 6 is a report on these two systems, and 
romes out in favor of BHD. The interested reader is referred to the 
two program system manuals for further information: 

^'Statistical Package for the Social Sciences,** by Norman H. Nie, Dale 
H. Bent, and C. Hadlai Hull. New York, McGraw-Hill, 1970. pp. 208- 
2kk. 

''BMD Biomedical Computer Programs, X-series Supplement,** W. J. Dixon, 
Editor. Berkeley, University of California Press, 1970. pp. 90-103. 

Given the report, we have elected to use the BMD X72 program. We 
expect to try several ways of filling the major diagonal, and several 
methods of rotation to see if the structures that emerge are substan* 
tially different, and if so, whether they lead to different insights. 

It will be particularly interesting to trace the factor stucture as a 
function of time as the curriculum matures. 

Task ID. Develop a statistical dftsign. 

The descriptive statistics to be used have been described previously. 
The complexity of the information system and the measures of perfor- 
mance now make the design of the inferential procedures more complicated 
than had been thought. Analysis of variance and covariance will be 
part of the scheme, but more work needs to be done before a set of 
tests can be determined. 
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CONCLUSIONS 



The following conclusions have arisen from this study: 

1. Evaluation procedures are part of the essential genetic material of 
a program of curriculum development. They also serve as a vital topic 
of dialogue among participants. 

2. Course descriptions must be couched, at least partly, in terms of 
intended effects upon students, and how the effects are to be measured. 
This information will be used to modify Tables 1 and 5 as it emerges. 

3. An analysis must now be made of costs of this plan, snd sources of 
support found for implementing it. 

^. It is possible to recruit program students into the process of curr- 
iculum evolution; although not all of them show equal involvement, some 
of them become deeply involved. 

5. It is very difficult to generate satisfactory performance & behavior- 
al objectives for the Icind of program being developed here. Yet a start 
has to be made. Any list of such objectives must be viewed as subject to 
revision as experience, advice, and interaction accrue. 

6. There are important kinds of desired outcomes that cannot necessarily 
be articulated in readily measurable form, and those that are measurable 
may come to receive inordinate amounts of attention if care is not used. 

7' At least some faculty members are willing and able to think in behav- 
ioral terms for their courses and the program as a whole, although they 
tend to express reservations such as those in 5 and 6 above. 

8. Generally speaking, students have high expectations of the program. 

9. For the most part, students' ratings of their actual experience are 
not far from their desired ratings. Although generally somewhat lower 
than ••desired," "actuaT* ratings were higher in many cases. 

10. From the several sections of one of the courses, where objectives, 
structure, and textbooks were the same. It appears that students* rat- 
ings of a course may depend to a significant extent on the instructor. 
Hence, the procedure we have been using, of recruiting carefully select- 
ed faculty members, is preferable to asking departments for courses, or 
specifying courses and having them manned ''as available.'' 
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APPENDIX 1 

UNDERGRADUATE STUDIES IN COMPLEX PROBLEM SOLVING 



PRELIMINARY DATA SHEET 



To enable tis to understand the educational plans of applicants to 
this program, and to provide counseling In as useful a manner as 
possible » we need some preliminary Information. Thxs will also 
help In the conduct of the personal Interview. In turn» the questions 
may give you some Insight Into our approaches. 

Although you will find requests for grade averago and other numerical 
figures, we place emphasis on the qualitative factors and attitudes 
of people » which cannot be accurately measured through objective 
tests alone. Consequently, your candid and considered responses to 
the data sheet are important and %^11 be appreciated. 

You may take the form home to complete when you can. Admittance to 
a course Is by consent of the Instructor after this form and a per- 
sonal Interview have been completed. The course Is part of a program 
In complex problem solving and we are looking pi»rticularly at what 
our students have to say. So please feel free to be as Imaginative 
and free*wheellng as you wish. 
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THE RAW FACTS ARE ; 
My name Is: 

The date I completed this form Is: 
for quarter, 197^. 
My permanent address Is: 

My mailing address Is: 



You may telephone me at: 



(area code) phone 
My age Is: 

Currently I am: single divorced 

married widowed 



I work hours/week after school because: 



6Q 



2. 



EDUCATIONAL SITUATION 

I am classified as: 

Fr«8h« Soph. Jr. Sr. 

Hy major Is: 

I muld nice my major to be: 
C^^dments: 



I have previously attended college or University 

List If more than one: 

I completed high school In 



name city and year graduated 

I took the SAT and scored & 

math verbal 

I vould like the opportunity to take It over: and/or 

yes no 

because : 
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-3- 

Thus far, I have an overall GPA of 

«nd a ^ GPA In my major. 

I would like to comment on my GPA, as follows: 



I have received honors or exceptional recognition for my past 
work, such as 



62 



-4- 

I have been Involved in the following extra currlcular service activities: 



I have gained positive or negative recognition in my past schooling which 
I would like you to know about: 
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-5- 



Crltlcally evaluate your previous education. 



My main motivation In joining the Problem Solving Program Is 



6k 



I am Interested In taking the following courses in the "momentum program" 
because : 



I am considering attending the following professional school (s) because: 
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1 



-7- 



1 think this program, in studying ^.he dimensions of complex problems, 
may be worth pursuing because: 
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ERIC 



APPLICANT SELF EVALUATION FORM 

(Please add any additional materials or papers you feel you want to use) 
What qualities of personality and Intellect do you like and dislike about 
yourself? What qualities do you most admire In others? 



If you were to select application materials about yourself to submit for 
a position you wanted, what would you choose? and which would you want 
the selection committee to weigh most heavily? which would you want dis- 
counted and why? 
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-2- 



What experiences have had the most significant impact on your growth 
and how have they affected it? 



If the above answers do not fully express what you would like to say 
about yourself, please use the space below to formulate what you feel 
to be more relevant or complete, in any way you wish. 
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APPENDIX 2 



Dear 

Enclosed you will find an evaluation form designed to Inform us about 
your Ideas and attitudes regarding your course experiences In the mo- 
mentum phase of the Creative Problem Solving Program, As we discussed 
In person, your cooperation In this evaluation process Is essential, and 
was agreed to when you were admitted to the course. Please remember 
that this Is not yet a fully actualized program. The courses represent 
first year offerings. Intended as a start-up set of experiences , rather 
than a finished program. To allow the University to benefit most from 
having done this and to help us design the program for the future we 
need careful and conscientious feed-back from you. 

The evaluation form is in three parts. Please complete the form as it 
is arranged, starting with Part I, going on to Part II, and ending with 
Part III. Do not let the reverse numbers of the questions in Part II 
disturb you. The numbering will expedite our scoring procedure with a 
Computer program. 

You should be able to complete the form in about two hours. Before 
starting, you might like to look at Part I only , and consider the questions 
carefully for a couple of days before sitting down to write the entire 
form. Once you begin to write your responses, please finish the entire 
evaluation in order, at one sitting. 

Your answers will be kept confidential to whatever degree you request. 
Kindly indicate, on the first page, your feelings regarding this issue. 
In any case, grades will be turned in before we examine these responses. 

Everyone must return the forms on or before Monday, March 29, 1972 in any 
of the following ways: In person, by regular post, or through the campus 
mail (free). Send completed forms to: 

Nancy Arnold Towers 

Architecture and Urban Planning - 1118 
UCLA, Los Angeles, CA 9002A 

In the near future, we will hold a meeting for the students and faculty 
who have been in the program. Your invitation to the meeting and information 
on time, date, and place will be handled either through a cooperative net- 
work of phone calls or postcards, and information will be posted on my door. 

Thank you for your interest, intelligence, and help inall these matters. 
It is proving very rewarding to work with you, and we look forward to 
further collaboration. If you have any questions or comments, please con- 
tact me at xS4994, or leave a message at x53791. 

Good luck! 

70 

Nancy Arnold Towers 



CREATIVE PROBLEM SOLVING CURRICULUM 
STUDENT EVALUATION FORMS 



COURSE(S) TAKEN; 

1971 

'^'^'^'^^^^^ PROFESSOR(S) : 



ZIP 

TELEPHONE: ^ycg. 



MAJOR: YEAR: PR, SOPH, JR, SR. 

(CIRCLE) 



If additional space is required, for any part of the questionnaire, 
please use back of page. , 



( 
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PART I 



The rationale for the questions in Part I is as follows: Evaluators, 
as well as instructors , hold particular perspectives of the skills re- 
quired for graduate training. Moreover, each has his own model for 
problem solving effectiveness. So, while we may be able to evaluate 
students against skill criteria which fit our models, we cannot des- 
cribe the meanings students attribute to their learning activities. 
We want to try and remedy this by monitoring the growth processes or 
changes you feel you have undergone as a result of your participation 
in this program. It should not only be a collection of stresses. 
Nor ought it to be a passive acceptance of all that occurred. We 
hope you will discuss the way the classes proceded, pointing out the 
areas of differences (or problems) and areas of strengths as you 
experienced them. With this introduction in mind, please answer the 
fol lowing : 

1) What are your bases for judging the course? i.e. in highly personal 
terms, list the specific criteria (as many as you wish) that you would 
use in judging the meaning this course seems to hold for your life. 

2) Evaluate this course on the personal criteria you listed. 

a) In what way was the course most successful? 

b) In what way was the course least successful? 
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PART II 



The preceding section involved a highly personal "think-piece'\ designed 
to organize your feelings and ideas in writing and create a mind-set 
for your replies. Now we have a chance to see whether and if so how, 
your perspective converts to ours. 

Part II requires your thoughtful responses by circling or numbering 
the appropriate answers* Please read the directions carefully befo re 
proceeding. 

Also, please omit those questions that are preceded by a check mark (X) . 
Thank you. 
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PART I I I 



The following questions will require w rttten answers. Please answer 
them as fully as you v/ish, keeping in nind that we are looking for 
honest, direct, and thoughtful replies. Please don't try to interpret 
what you think we want to hear. There are no sanctions (for or against) 
nor prescriptions for a "right" answer. They are asked in order to 
provide Information on how to better organize the program; design the 
next stages; and build a more responsive curriculum. If additional 
space is required, please use additional paper. 

Thank you. 



1. Are you pleased you took the course(s) that you did? 

2. Do you want another "follow up" course based upon the learning exper- 
iences in this course? 

What design? 

With which Professor (s) on campus? 



3. Are you interested in taking more courses of the kind this program 
promises? WKicb courses and why? 



^. Are you willing to g'-» "def'gn time" to the creation of learning 
experiences for the program? 

5. H.3ve your professional plans or expectations changed or been modified 
at all as a result of your experience in this course? If yes, how or why? 
If not, why not? 
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6, What kind of work do you intend to do upon graduation? 



7. Do you see this program as benefitting you In your long range plans? 
If so, how? If not, why not? 



8. Have you changed your mind regarding the size and complexity of problems 
you find challenging? If so, why? and if not, why not? 



9. Communicating information about the program, on a campus-wide basis, 

Is a continuing problem for us. Can you suggest feasible methods for changing 
this? Are you interested in helping In this process? 



lu 



We want to recruit outstanding faculty on campus into the program. Please 
list the names of those members of the faculty (from all departments) whom you 
most esteem, would enjoy working with, and who might be interest J in our program. 



Thank you. 
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APPENDIX 3: TABLE 1 
Key to Compu ter Printout of Student Course Evaluations 
Table I; response codes, responses, and computed, desired, actual, and 
total frequencies and percentage frequencies: 

A. code; response type code number 

B. FQA; total response frequency of an ''actual'* response 

C. % FQA; percentage total response frequency of an "actual" response 

D. FDQ; total response frequency of a "desired" response 

E. % FDQ; percentage total response frequency of a "desired" response 

F. FQT; total response frequency of both an "actual" and a "desired" 
response 

G. % FQT; percentage total response frequency of both an ^xtual" and 
a "desired" response 

H. Response; response type syntax 

Table II; course question statistics summary: 

A. ADM DF; (actual-desired) mean difference 

B. CODE; response type code nui.:ber 

C. FREQ; response frequency 

D. % FREQ; percentage response frequency 

E. MN ACT; "actual" response mean 

F. MN DES; ''desired" response mean 

G. MSE ACT; "actual" response mean standard error 

H. MSE DES; "desired" response mean standard error 

I. STD ACT; "actual" response standard deviation 
J. STD DES; "desired" response standard deviation 
K. VAR ACT; "actual" response variance 

L. VAR DES' "desired" response variance 

Table III; student statistics summary: 

A. AR; "actual response 

B. ARM; "actual" response mean 

C. ARMSE; "actual" response mean standard error 

D. ARSTD; "actual" response standard deviation 

E. ARVAR; "actual" response variance 

F. CODE; response type code number 

G. % DIS; percentage ("actual"-"desi red") response disparity 

H. DRi "desired" response 

I. DRM; "desired" response mean 

J. DRMSE; "desired" response mean standard error 

K. DRSTD; "desired" response standard deviation 

L. DRVAR; "desired" response variance 

M. FREQ; response frequency 

N. % FREQ; percentage response frequency 

0. MR; mean response 

P. MRDIF; mean ("actual"-"desi red") response difference 
Q. Q; question number 

R. RD; ("actuar'-"desired") response difference 



IV. Sunvnary: 



A. QUESTIONS; question syntax 

B. # QUEST; question number 

C. MN ACT RESP; mean "actual" response 

D. MN A-D RSP DF; mean ("actuar'-"des i red") 

E. MN DES RESP; mean "desired" r isponse 
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At present, two general purpose statistical packages with factor 
analyses modules on the CCN IBM 360/91 are available: 

1. Statistical Package for the Social Sciences (SPSS, subprogram 
FACTOR). 

2. Biomedical Computer Programs (BMD, subprograms BM03M and BMDX72). 
Unfortunately, the more powerful SPSS factor analysis module has a 

history of confirmed and suspected numerical instabilities. Consequently, 

FACTOR generated results are highly suspicious in terms of validity and 

reliability. The SPSS designers currently claim the factor analysis module 

has been stabilized. However, this claim has yet to be established by 

independent tests with any acceptable degree of assurance. Some UCLA 

investigators feel subprogram FACTOR is now reliable while others claim 

Internal problems still exist. Therefore, in considering the probable 

high expenditure of computer funds and man-hours and the sensitivity of the 

decision-making requirements, the risk in utilizing subprogram FACTOR does 

not seem warranted. These circumstances are indeed unfortunate because 

SPSS control syntax and general structure have been designed with the 

non-computer-oriented user in mind. 

While the BMD package is not as sophisticated as SPSS in terms of 

control syntax, the subprograms have a long history of numerical stability. 

Consequently, BMD03M and BMDX72 may be used with a high degree of assurance. 

As far as numerical procedures and methodology are concerned, BMD and SPSS 

are equivalent; therefore, the user is not making a trade-off between 

reliability and procedural flexibility. For a brief description of the 

mathematical treatment, control syntax, and general rules of usage for 

FACTOR, BMD03M, and BMDX72 refer to sections two, three, and four respectively. 
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A third choice is possible. With consideration given to the 
specific requirements of the user agency, the development of a separate 
factor analysis module may be feasible. If factor analysis only is 
considered as the key decision-making statistical method, then only the 
BHD or SPSS modules need be considered. However, if factor analysis will 
be only one of a series of methods, then a scheme or general system design 
has to be considered. Highlights of this general system are as follows: 

1. Job Control Language (JCL) 

2. User procedures and docu^^entation 

3. User control syntax 

4. File generation and manipulation 

5. Data gathering 

6. Preliminary data preparation 

7. Preliminary or marginal statistics 

8. Descriptive statistics 

9. Inferential statistics 
10. Report generators 

n. System transportability to a non-IBM configuration 
12. Source code availability 
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GENERALIZED STATISTICAL 
EVALUATION SYSTEM FLOW 
CHART 



C 



qUESTION- 
KAIBES 



^START^ 



\ DATA I 



XNTEBMEDIATS 
DASA FILB 



BEQUXB£D 
X!n!ERMEDIAT£ 
nAXA FILE 



INTERMEDIATE 
DATA FILE 



BEQUIHED 
INTERMEDIATE 
DATA FILE 




DATA 

PREPARATION 
MODULE 




MARGINAL 
STATISTICS 
MODULE 




DESCRIPTIVE 
STATISTICS 
MODULE 




INFERENTIAL 
STATISTICS 
MODULE 



/ RAW \ 
"\ DATA / 



PREPARED 
DATA FILE 





DATA 




PREPARATION 












GENERATED 
INTERMEDIATE 
DATA FILE 



MARGINAL 
STATISTICS 
REPOR' 





GENERATED 
INTERMEDIATE 
DATA FILE 



DESCRIPTIVE 
STATISTICS 
REPOR' 



GENERATED 
INTERMEDIATE 
DATA FILE 








INFERENTIAL 






STATISTICS 






REPORJ^--^ 







SUMMARY 
MODULE 




GENERATED 

SUMMARY 
DATA FILE 



SUMMARY 
REPORT 
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With any program or system usage, very careful attention has to 
be given to the IBM 360 OS Job Control Language (JCL) as it applies to 
CC.'s configuration. Nothing can be more frustrating to the new or 
inexperience user than trying to interact with the computer via a JCL 
scheme that is incomplete, not documents, or poorly designed. 
Consequently, the user must be provided with a transparent JCL scheme 
that is easily used and fault free. This requirement concerns both the 
use of existing packages and a tailored system. 

User-oriented documentation describing all aspects of program or 
syster: usage is of paramount importance. Such a document should include 
the following: 

1. JCL scheme. 

2. Error messages and Interpretation. 

3. Data management. 

4. File management. 

5. System order and flow. 

6. Module descriptions. 

7. General and specific system description. 

A prime cause of program or system failure with respect to the 
user is usually caused by a lack of or poorly edited documentation. 
Normally, the "cookbook" approach is the safest course. 

Data gathering and preliminary preparation is normally the first 
required action by the user. The reliability of any statistical methodology 
is directly proportional to the care taken by the user in the generation 
of his data. 
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Preparation would include the following: 

1. Identification. 

2. Error scanning. 

3. Sample selection. 

4. "Massaging" 

In many cases, computer funds and personnel time are optimized 
by a well designed use of file generation and usage. Most available 
statistical packages allow the use of intermediate data structures 
rather than starting with raw data in every case; a tailored coni iguration 
can be easily designed with the same facility. The following should be 
considered with reference to file management: 

1. Identification 

2. System/ program interface. 

3. Structure. 

4. Device type. 

5. Security. 

The choice of marginal, descriptive, and inferential statistics 
depends entirely upon the user requirements. In this report, only factor 
analysis has been given consideration; however, other equally powerful 
methods are available. The particular methods chosen should allow the 
user to examine intermediate results before continuing to the next 
computational phase. This procedure gives the user a chance to detect 
and deal with subtle errors or suspicious results. 

In considering the class or user involved and the application, 
generated reports must be carefully constructed in such a fashion as to 
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provide maximum information with relatively little paper work. This 
approach offsets the possibility of data "overkill." Unfortunately, the 
report format structure of BHD leaves much to be desired; SPSS is only a 
little more acceptable. Consequently, tailored reports will probably have 
to be developed no matter what pakcage is chosen or what system is developed. 

If either BHD or SPSS is chosen, the question of source program 
availability becomes critical. In general, SPSS source code is not 
available to the general public. However, this department possesses the 
most current source modules for all available BMD subprograms. Although 
source code availability is less critical with respect to a generated 
system, considerable funds and time may be conserved by utilizing whatever 
available source programs may be easily Integrated in the design phase. 

System cransportabillty need only be considered if implementation 
upon a machine configuration other than CCN's is required. Normally, the 
effects fall into the following areas. 

1. JCL. 

2. File management. 

3. Source code compatability. 

If the system is to remain resident at UCLA, full advantage can be 
taken of CCN's unique characteristics. However, if the system is also to 
be Implemented elsewhere, overall design might well be generalized to a 
greater degree initially to avoid need for costly modifications by others 
in the future. 

In summary, BMD03M or BMDX72 is more attractive than FACTOR in 
terms of numerical stability. However, the SPSS package has a more 
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flexible control syntax than BHD. With all things considered, the 
generation of a system as briefly described may. In fact, be the wisest 
choice. For a brief description of the SPSS and BMD subprograms 
availability, refer to sections five and six respectively. 
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APPENDIX 5 



nitrFSiHoI^fo^J.'i" Education Measurement and Evaluation Kit. 

Fie d Edition 1972" prepared by tiie staff of tiie Higher Education 

Sc'hiot'IrEdi^jn.'"'" ''"'^ °' Evaluation. UCLA Graduate 

Jhe^Kir* "-ationales and norms for tiie various scales are provided in 
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EDUCATIONAL BBIEFITS 




Vocational 




DiTections: In thinking over your experiences in college ip to 
now, to vfcit extent do you feel that you have made progress or 
been benefited in each of the following areas? Indicate your 
rating by placing a check in the space under the nuiiber vAiich 
best describes (a) the ACTUAL and (b) the PREFERABLE amounts 
of progress or benefit. 


Key: 5 = Very much 
4 = Quite a bit 
3 = A moderate amount 
2 = Some 

1 = Little or none 


ACTUAL 


PREFERABLE 


5 4 3 2 1 


5 4 3 2 1 


1. Background and specifica* 
tion for further education 
in some professional, 
scientific or scholarly 
field. 




2. Bases for iinproved social 
and economic status. 




3. Vocabulary, teiminology, 
and facts in various fields 
of knowledge. 




4. Vocational training- -skills 
and techniques directly 
applicable to a job. 
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EDUCATIONAL BENEFITS 




I^nanistic 




Directions: In thinking over your experiences in college up 
to now, to what extent do you feel that you have made progress 

^ItS^K?"? -^^ ^ !^ following areas? Indicate your 
rating by placing a check in the space jnder the number which 
best describes (a) the ACTUAL and (b) the PREFERABLE amounts of 
progress or benefit. 


Key: 5 = Very much 
4 = Quite a bit 
3 = A moderate amount 
2 = Some 

1 = Little or none 


ACTUAL 


PREFERABLE 


5 4 3 2 1 


5 4 3 2 1 


1. Awareness of different 
philosophies, cultures, 
and ways of life. 




2. Broadened literary ac- 
quaintance and apprecia- 
tion. 




3. Aesthetic sensitivity- - 
appreciation and enjoyment 
of art, music, drama. 




4. Writing and speaking- - 
clear, correct, effective 
communication . 
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HXJCATICML BENEFITS 




Critical Thinking 




Directions: In thinking over your experiences in college up to now, to v^at 
extent do you feel that you have made progress or been benefited in each of 
the follovdng areas? Indicate your rating by placing a check in the space 
under the nunber >Aich best describes (a) the ACTUAL and (jb) the PREF^ABLE 
amounts of progress or benefit. 


Key: 5 = Very much 
4 - Quite a bit 
3 « A moderate amount 
2 = Some 

1 - Little or none 


ACTUAL 


PREFERABLE 


5 4 3 2 1 


S 4 3 2 1 


1. Develop intellectual curiosity-- 
interest in discovering and under- 
standing problems. 




2. Develop intellectual honesty and 
objectivity- -seeking the best 
possible ansvier, accepting the con- 
sequences of the inquiry, and re- 
porting the findings accurately 
and without intentional distortion. 




3. Develop open-mxndedness--va.thholaLng 
judgment, examining contraxy views, 
changing one's opinion in tiie 
licht of the facts. 




4. Develop a desire for persistence 
and order- -searching for systematic 
relationships even \ihen a solution 
is not imnediately clear. 




5. Develop the ability to define problems- - 
breaking into elements, formulating re- 
lationships and qtestions which are 
testable. 




6. Develop the ability to recognize 
assunq[)tions- -stated, instated, 
sipported, and irrelevant. 




7. Develop the ability to select infor- 
mation relevant to the problems- - 
distinguishing reliable infoimation 
and sources. 




8. Quantitative thinking --understanding 
concepts of probability, proportion, 
margin of error, etc. 




9. Develop an understanding of the nature 
of science, experimentation, and theory. 
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EDUCATIQN/U. BENEFITS 




Bjonan Relations 




Directions : In thiiiking over your experiences in college vp to 
now, to vrfiat extent do you feel that you have made prpgieoS or 
been benefited in each of the following areas? Indicate your 
rating by placing a check in the space wder the nunfcer \^ich 
best describes (a) the ACTUAL and (b) the PREFERABLE amounts 
of progress or benefit. 


Key: 5 = Very much 
4 = Quite a bit 
3 « A moderate amount 
2 = Some 

1 = Little or none 


ACniAL 


PREFERABLE 


5 4 3 2 1 


5 4 3 2 1 


1. Personal development-- 
understanding one's 
abilities and limitations, 
interests, and standards 
of beh£Hdor. 




2. Development of friendships 
and loyalties of lasting 
value. 




3. Appreciation of individual- 
ity and independence of 
thought and action. 




4. Social developnient--e3qf)eri- 
ence and skill in relating 
to other people. 




5. Tolerance and inderstanding 
of other people and their 
views. 




6. ^>preciation of religion-- 
moral and ethical standards. 
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IWOLVBfENT IN C^MFUS REFGSMS 

Directions: Check (X) each activity you have engaged in during 
the past year. 



During the past year: 

1. I voted in a caapus election 

Z. I canipaigned for certain issues or people in a campus 
election 

3. I served on a student or college conmittee dealii^ with 

some caapos refbnn- -rules, courses, etc. 

4. I participated in student government 

5* I vfas an active raenter of a campus service organization 

6. I passed out literature on some canpis issue 

7* I contacted an administrative officer about a canpus issue 

8* I siqyported or participated in a groijqp protest over some 

ca]i|>us problem 
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STYLE OF LEARNING: ACADBOC 



Directions: Please check OC) each activity you engaged in duri 
the most recent "typical" \tcek. 

During a typical week : 

1. I participated in a class discussion. 

2. I h^d a conversation, lasting a half -hour or longer, 

with one or more of my professors. 

3. I discussed with other students for an hour or longer 

the siAject-natter of one or more of my courses. 

4. I spent a concentrated period of time- -three hours or 

longer without intemption- -studying for one of my 
courses. 

5. I studied at least four hours or longer during the 

weekend. 

6 . I read a book related to one of my courses but that 

was not an assigned reading for the course. 

7. I spent five or more hours writing papers. 

8. I spent sane time just browsing in the library or 

bookstore. 

9. I participated in a research project. 

10 • I spent five hours or more looking references in the 

library and taking notes. 
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STYLE OF LEARNING: EXPERIENTIAL 

Directions: Oieck (X) each activity you have engaged in during the 
past tem. 

During the past tern: 

^_ 1. I participated in a sensitivity groip. 

2. I became well acquainted, personally and socially, with 

a student vdiose race is different than mine. 

3. I became well acquainted with a foreign student. 

4. I did sone woik witii children or parents in a neighborhood 

different frcxn mine. 

5 . r had one or more long and private conversations with a 

professor or counselor. 

6. I was in an infoimal social Mthering ^re students 

and &Knilty mrai)ers came to mm one another personally 
(not just superficially). 

7. I had the experience of being in a marathon encounter 

grovp. 

8. I helped a student 1*0 was having difficulty in a course. 



t 
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Key: HS = 


higjily satisfied 


S = 


satisfied 


N = 


neutral 


D = 


dissatisfied 


HD = 


highly dissatisfied 



VARIBIY OF DBnUCriCKAL AM) LEARNING EXPERIENCES 

Directions: For each of the following instiuctional or learning experiences, 
please indicate, by placing a check QC) in the appropriate colunn, (a) if you 
have had the experience, (b) if you have not but wuld like or not like to have 
the experience, and (c)'if you have had the experience, the degree to which it 
was satisfying to you, using the key on the left. 



mVE »U) THIS 
EXPERIBO 

yes 



1. Snail class with instructor- 
led discussions 

2. Large lecture classes 

3. Lecture class with scheduled 
discussions sections 

4. Video-taped lectures 

5. Team teaching-two or more in' 
structors teaching a course 

6. Group projects, as part of 
course woric 

7. Individual research » as 
part of course work 

8. Groip research, as part of 
course work 

9. Student-led discussion groups 

10. Courses involving connunity 
e^riences 

11 • Part or all of the course 
work conducted off-campus 

12* Interdepartmental course 
(course involving instruc- 
tors or materials fron more 
than one department) 

13. A laboratory course 

14. Independent st\xdy 



WOULD LIKE 
TO HAVE THIS 
EXPmiENCE 

yes nd 



DEGREE OF 
SATISFACTION 



HS S N 
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NOIABLE BCFERIENCES 



What stands out in your mind so far about your college experience? 
(Check as many as apply Q 



1. Living amy from home. 

2. Informal discussions with other students. 

3. Participation in sports. 

4. Being involved in some extra-curricular activity such as 

msicp drama, politics, etc. 

5. The e3q>erience of leadership in some campus activity. 

6. Particular professors vflio have taken a personal interest 

in me. 

7. Particular professors vtoo encoux'aged me in ny work. 

8. Particular courses tiiat opened up new interests for roe. 

9. Some lectures that were particularly stimulating. 

10. Realizing what the demands of good scholarship really are. 
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AREAS AND AGBITS OF CHANGE DURING COLLEGE 



Directions: For each of the following areas, please indicate how much you have 
changed while in college, and how mudi inflience each of the "agents'* have been 
in that change • Place a check (X) in the space below the nunber vAich most 
accurately reflects (A) the overall amount of change, and (B) the amouit of in- 
fluence each "agent" has had in bringing about that change. 



Key: 3 = very nuch 
2 = sone 
1 = little/none 



A. OVERALL 
CHANGE 



AREAS OF CHANGE 



3 2 1 



S. AGENTS OP CHANGE 

Fellow General Wbrk ex- 
Courses Faculty students maturity periences 



3 2 1 



3 2 1 



3 2 1 



3 2 1 3 2 1 



1. Development of a 
world view and 
personal philoso- 
phy 

2. Development of an 
interest in new 
fields of learn- 
ing 

3. Development of 
general thinking 
skills 

4. Development of 
an identity and 
sense of self- 
confidence 

5. Developnent of 
social skills 



6. Development of 
career plans 
and skills 



7, Development of 
a positive 
attitude to- 
wards this 
college 



GQIBRAL SATISFACriGN 
WriH OQLI£GE 



1. How well do you like college? (Check One) 

_ 1. I don't like it. 

2. I am more or less neutral about it. 

3. I like it. 
4. I am enthusiastic about it. 

2. If you could start over again, would you go to the same college 
you are now attending? (Check One) 

1. No, definitely 

2. Probably no 

3. Probably yes 

___4. Yes, definitely 

3. Regardless of any vocational benefit , college may have for you, 
do you think that being in college at this time in your life is 
a very ijujortant and beneficial experience? (Check One) 

1. Definitely no 

2. renerally no 

3. dener J.iy yes 

4. Definitely yes 



133 



INTEILHTU/U. GRIENIATIQN 




Directions: life all have different preferences and personal characteristics. 
Beside each item, on the ri^t, please indicate how characteristic the state- 
ment is of you. (Response key: VM « very much, QB « qxrite a bit, S « some, 
« not at all). 


1. Appreciate original work 


VM QB S MA 


2. Enjoy dealing with new or strange ideas 


^ — — — 


3. Like original research work 


— — — — 


4. Prefer to draw iny own coiKlusions 


— — — — 


5. Like to think critically about theories 




6. Lii.e people interested in ideas 




7, Often question authoritative opinion 




8. Experiment with ideas 


— — — — 


9. Bijoy seeking knowledge for its own sake 




10. Het-ve diverse interests 




11. Play with a variety of ideas 




12. Use my imagination 




13. Try to connect seemingly unrelated ideas 




14. Examine ideas regardless of their practicality 




15 . Discover new ways of doing things 





CRITICAL THINKING ORIENTATION 

SS^eJ^ ifj^Hr ^^S^^J preferences and perscaial characteristics, 
m^i n?vi 'r? ^ "^^^^ Pf^® indicate how characteristic the state 
iot ft^aU) ^ " "^"^ ' 'l"^^^ * ^i^' S = some, 



1. 
2. 
3. 

4. 

5. 



Enjoy solving long conplex problems 

Enjoy detecting faulty reasoning 

Enjoy searching for systematic relationships 
even vtfien a solution is not inmediately 
^jparent 

Enjoy examining contrary views 

Think about the accuracy of infbimation 
I receive 

Withhold iny opinion until carefia consi- 
deration of the facts and opposing ppinions 

Examine statements critically 

Question people when they present new ideas 

like to conteirplate the future of society 

10. Argue different sides of an issue 

11. Recognize the assimptions behind an argument 

12. Like to seek the best possible answer, 
even if it takes quite a long time 

13. Enjoy discovering new problems to think 
about 



7. 
8. 
9. 



VM QB S NA 
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SCIENTIFIC CRIENTATICN 

Erections: Vfe all have different preferences and personal characteristics. 
Beside each item, on the right, please indicate how characteristic the state- 
ment IS of you. (Response key: VM - very much, QB = quite a bit. S = some, 
■ not at all). * 

VM QB s m 

1. Enjoy scieice and mathematics 

2. Bijoy discovering how things woric 

3. T-ooking at scientific displays 

4. Solving puzzles 

5. Like to formulate ray own hypotheses about 
things 

6. Like to take things apart to see how they 
work 

7. Like to conduct ray own experiments 

8. Bijoy observing things and thinkii^ about 
how they work or occur 
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FEELINGS ABOUT SELF 






Directions: We all have different preferences and personal characteristics. 
Beside each item, on the right, please indicate how characteristic the state- 
ment is of you. (Response key: W = very much, QB = quite a bit, S = some, 
MA = not at all) . 






VM QB S 




1. 


When people say nice things about me, I find 
it difficult to believe they really mean it. 






2. 


I look on most of the feelings and inpilses 
I have tov^ard people as being quite natural 
and acceptable. 






3. 


I feel different from other people. I*d like 
to have the feeling of security' that comes 
rran Knowing I'm not too dirfierent from 
others. 


— — — 


— 


A. 


I am frequently bothered by feelings of 
inferiority • 






5. 


In order to get along and be liked, I tend 
to be \4iat people expect me to be rather 
than anything else • 


— — — 


— 


6. 


I seem to have a real inner strength in 
nanaiing tniiigs . 






7. 


Very often I don't tty to be friendly 
with people because I think they won^t 
like me . 






8. 


I feel that I'm a person of worth, on an 
equal plane with others . 






9. 


I live too much by other peoples* 
standards . 






10. 


When I have to address a group ^ I get 
self-conscious and have difficulty saying 
things well . 
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FEELINGS ABOUT OHIER PEOPLE 










Directions: We all have different preferences and personal character- 
istics. Beside each item, on the right, please indicate how character- 
istic the statement is of you. (Response key: VM « very much, QB = 
quite a bit, S « some, NA ■ not at all). 






VM 


QB 


S 




± . 


1 rina it Jiaiti to talk inatn people wio hold 
qpinions qmte different frc?^ ny own. 


— 


— 


— 


— 


2. 


I find it exciting to meet people quite 
oitierent trom nyseli. 


— 


— 


— 


— 


3. 


I can become so absorbed in the woxk I'm 
doing tnat it doesn't bother me not to have 
any intimate friends. 


— 


— 


— 


— 


A 


1 nave round that people have to be pretty 
much like me if vwe are going to strike 
a friendship. 


— 


— 


— 


— 


5. 


There are few times when I ccmpliment people 
for their talents or jobs they^ve done. 










6. 


I try to get people to do what I want them 
to do, in one way or another. 










7. 


There^s no sense in coapromising. Mien 
people have valves I don^t like, I just don't 
care to have much to do with them.. 










8. 


I enjoy doing little favors for peq)le even 
if I don't knoM them well. 










9. 


I enjoy nyself post when I'm alone, away 
from other people . 










10. 


I can be friendly with people \itio do things 
of which I don't approve. 











c 
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FEELINGS ABOrr THE FUTURE 



Directions: Below you will find sone pairs of 
statements. Kead each statement in the pair, 
then circle the letter of the pair that you 
most nearly agree with. Even though it may be 
difficult, please make a choice. On the right 
indicate how sure you feel about the choice 
you made. 



Key: VS very sure 

MS » moderately sure 
NS ■ not sure 



VS MS NS 



1. A I have off^n found that ^diat is going 

to happen will happen. 

B Tiustiqg to fate has never turned out 
as well for me as makii^ a decision to 
take a definite course of action. 

2. A Hhat happens to me is my own doing. 

B Sometimes I feel that I don't have enough 
control over the direction ray life is taking. 

3. A Yhen I make plans, I am almost certain 

that I can mke them WDxk • 

B It is not always wise to plan too far 
ahead because many things turnout to be 
a matter of good or bad fortune anyhow. 

4. A In case, getting what I want has little or 

nothing to do with luck. 

B Many times we might just as well decide what 
to do by flipping a coin. 

5. A Many times I feel that I have little influence 

over the things that happen to me . 

B It is inpossible for me to believe that chance 
or luck play an iii|)ortant role in vsy life • 
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GEFmi VALUES AM) IDEOLOGIES 


As you read each of the wrds listed below, what sort of associations come 
to your ndnd? Are these inpionptu and iimiediate associations to the word (what 
It stands for in your mind) generally positive? generally negative? or mixed? 
After each word check your first reaction to it. 




Positive Mixed Negative Pdsitive Mixed Negative 


1. Sjpirituality 




16. Calculation 


2. Perspective 




17. Plannii^ 


3. Idealism 




18. Prudence 


4. Altruism 




19. Loyalty 


5. Mysticism 




20. Tradition 


6. Reverence 




21. Organization 


7. Freedom 




22. Obedience 


8. Spontaneity 




23. Self-Stocrifice 


9. Creativity 




24. Justice 


10. Piirticipation 




25. Salvation 


11. Pferceptiveness 




26. Righteousness 


12. Sensory awareness 


27. Worship 


13. Moderation 




28. Dignity 


14. Thoughtfulness 




29. Stdanission 


15. nexibility 




30. Revelation 

• 
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SOCIETAL miCRITIES 


The statonents below describe goals, purposes, and achie-vements that various 
people believe to be iii|)ortant in our society. What is your own view about 
the importance of these goals? And **at view do you think is held by the 
majority of people in our society? To the left of each statement, indicate 
your priority by checking the appropriate space: and to thi? right nf Mch 
statement, indicate vfliat priority you think the majority of people would 
give to the statement. 


YGUR PRIORITY 


MAJCRITY 


High Low 


High Low 


5 4 3 2 1 


5 4 3 2 1 


1. 


Pronote harmony and understanding 
among nations. 


2. 


Help each person develop and use 
his personal talents. 


3. 


Enhance the health, education, 
and velfare of all peq>le. 


4. 


Achieve a balance between man 
and the environment. 


5. 


£;q[ilore distant planets and 
outer space. 


6. 


Increase man's control over 
nature through technology. 


7. 


Develop the resources of the 
oceans. 


8. 


Advance civilization through 
science and reason. 


9. 


Help everyone attain material 
goods, economy, and status. 


10. 


Enrich life through spiritual 
renewal and dedication. 


^ 11. 


Insure the military and economic 
strength of the nation. 



HI 



c 



HXJCATIONAL PRICFITIES 

The statanents below describe goals, purposes, and achievements that various 
people believe to be inportant in hi^r education. What is your own view 
about the inportance of these goals for colleges and universities? And vtfiat 
view do you think is held by the majority of people in our society? To the 
left of each statement , indicate your priority by checking the appropriate space ; 
and to the right of each statement, inoicate >diat priority you think the najority 
of people would give to the statement. 



YOUR PRIORITY 
Hi^ Low 
5 4 3 2 1 

1. Prepare people for useful 

occupations. 

2. Provide leaders for society. 

^ 3. Transmit basic cultural values. 

4. Serve the economic needs of 

society. 

5. Develop a person^s self- 
awareness. 

6. Broaden the students^ perspective. 

7. Birich the students* life 

culturally. 

8. Seek and disseminate new knowledge. 

9. Help solve societal problems- - 

pollution, etc. 

^ 10. Serve as a critic of the society. 

11. Pnmiote understanding among 

different j^oples. 

12. Assist students in developing 

a personal philosophy of life. 



MAJORITY 
High Low 
5 4 3 2 1 
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